Top 100 Inference Engine Repositories

Back to Home

Ranking

Ranking	Project Name	Stars	Forks	Language	Open Issues	Description	Last Commit
1	vllm	85,853	19,208	Python	1952	A high-throughput and memory-efficient inference and serving engine for LLMs	2026-07-10
2	web-llm	18,334	1,321	TypeScript	132	High-performance In-browser LLM Inference Engine	2026-06-09
3	ml-engineering	18,333	1,169	Python	1	Machine Learning Engineering Open Book	2026-07-09
4	ds4	18,080	1,555	C	98	DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm	2026-07-03
5	MNN	15,624	2,381	C++	23	MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.	2026-07-09
6	Paddle-Lite	7,261	1,626	C++	51	PaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge (飞桨高性能深度学习端侧推理引擎）	2026-04-27
7	gemma.cpp	6,974	652	C++	22	lightweight, standalone C++ inference engine for Google's Gemma models.	2026-07-09
8	DALI	5,723	670	C++	195	A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.	2026-07-08
9	shimmy	5,621	541	Rust	11	⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.	2026-06-30
10	CTranslate2	4,561	498	C++	220	Fast inference engine for Transformer models	2026-07-03
11	Tengine	4,523	978	C++	243	Tengine is a lite, high performance, modular inference engine for embedded device	2025-03-06
12	TransformerEngine	3,420	769	Python	153	A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance wi...	2026-07-09
13	Rapid-MLX	3,241	381	Python	29	The fastest local AI engine for Apple Silicon. 4.2x faster than Ollama, 0.08s cached TTFT, 100% tool calling. 17 tool parsers, prompt cache, reasoning separation, cloud routing. Drop-in OpenAI replace...	2026-07-10
14	spiceai	3,019	209	Rust	370	Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.	2026-07-10
15	xDiT	2,651	327	Python	86	xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism	2026-07-09
16	AI-Engineering.academy	2,347	271	Jupyter Notebook	6	Mastering Applied AI, One Concept at a Time	2026-02-27
17	sonar	1,786	202	C++	77	Large-scale LLM inference engine	2026-07-10
18	openlake	1,782	296	Rust	69	OpenLake is a high performance storage engine for efficient LLM inference and GPU Training	2026-07-10
19	ai-performance-engineering	1,651	233	Python	3	Code, labs, and resources for O'Reilly AI Systems Performance Engineering: GPU optimization, distributed training, inference scaling, and full-stack tuning.	2026-07-06
20	uzu	1,646	62	Rust	0	A high-performance inference engine for AI models	2026-07-10
21	Genie-TTS	1,641	114	Python	30	GPT-SoVITS ONNX Inference Engine & Model Converter	2026-04-18
22	tokenspeed	1,573	189	Python	4	TokenSpeed is a speed-of-light LLM inference engine.	2026-07-10
23	xllm	1,458	255	C++	82	A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.	2026-07-10
24	airunner	1,312	99	Python	4	Offline inference engine for art, real-time voice conversations, LLM powered chatbots and automated workflows	2026-07-08
25	Jlama	1,291	159	Java	37	Jlama is a modern LLM inference engine for Java	2025-10-12
26	rtp-llm	1,265	230	Cuda	35	RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.	2026-07-10
27	openrouter-runner	1,251	125	Python	0	Deprecated inference engine	2025-09-06
28	FeatherCNN	1,228	275	C++	18	FeatherCNN is a high performance inference engine for convolutional neural networks.	2019-09-24
29	cache-dit	1,223	76	Python	87	A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.	2026-07-09
30	ezkl	1,213	211	Rust	12	ezkl is an engine for doing inference for deep learning models and other computational graphs in a zk-snark (ZKML). Use it from Python, Javascript, or the command line.	2026-02-20
31	Atomic-Chat	1,057	110	TypeScript	14	Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer.	2026-07-09
32	YOLOs-CPP	1,043	162	C++	1	Cross-Platform Production-ready C++ inference engine for YOLO models (v5-v12, YOLO26). Unified API for detection, segmentation, pose estimation, OBB, and classification. Built on ONNX Runtime and Open...	2026-07-04
33	nobodywho	1,020	70	Rust	28	NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.	2026-07-09
34	checkpoint-engine	965	91	Python	3	Checkpoint-engine is a simple middleware to update model weights in LLM inference engines	2026-07-04
35	ssd	965	76	Python	1	A lightweight inference engine supporting speculative speculative decoding (SSD).	2026-05-10
36	TinyChatEngine	956	100	C++	35	TinyChatEngine: On-Device LLM Inference Library	2024-07-04
37	ZhiLight	905	102	C++	5	A highly optimized LLM inference acceleration engine for Llama and its variants.	2026-03-18
38	tiny-vllm	893	61	C++	0	Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM	2026-07-02
39	emlearn	743	75	Python	16	Machine Learning inference engine for Microcontrollers and Embedded devices	2026-04-02
40	kronk	682	46	Go	6	Your personal engine for running open source models locally. Use Go for hardware accelerated local inference with llama.cpp and whisper.cpp directly integrated into your Go applications. Kronk provide...	2026-07-09
41	libonnx	650	114	C	16	A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support.	2026-07-07
42	atlas	584	83	Rust	25	Pure Rust Inference Engine	2026-07-10
43	tidy	577	43	Kotlin	34	Offline semantic Text-to-Image and Image-to-Image search on Android powered by quantized state-of-the-art vision-language pretrained CLIP model and ONNX Runtime inference engine	2024-03-28
44	WhisperS2T	575	76	Jupyter Notebook	31	An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine	2024-08-27
45	swama	570	30	Swift	34	High-performance MLX-based LLM inference engine for macOS with native Swift implementation	2026-06-08
46	qwen600	556	48	Cuda	0	Static suckless single batch CUDA-only qwen3-0.6B mini inference engine	2025-09-08
47	Anakin	537	135	C++	53	High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.	2022-09-23
48	openinfer	526	75	Rust	89	Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2	2026-07-10
49	VectorHub	524	134	Jupyter Notebook	1	Deprecated historical repo. Superlinked now develops SIE, a self-hosted inference engine for embeddings, reranking, OCR, extraction, and document processing.	2026-07-06
50	audio.cpp	480	39	C++	5	An all-in-one, pure C++ inference engine for audio models, powered by ggml. Supports TTS, STT, VAD, voice conversion, music generation, and more, with highly optimized performance. No Python dependenc...	2026-07-09
51	OpenArc	479	40	Python	8	Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.	2026-07-06
52	simple-llm	478	36	Python	0	~950 line, minimal, extensible LLM inference engine built from scratch.	2026-01-09
53	hipfire	470	48	Rust	55	RDNA-native LLM inference engine in Rust.	2026-07-10
54	crabml	470	45	Rust	24	a fast cross platform AI inference engine 🤖 using Rust 🦀 and WebGPU 🎮	2025-01-04
55	dotLLM	467	52	C#	178	LLM inference engine written in .NET	2026-04-30
56	ntransformer	465	19	C++	1	High-efficiency LLM inference engine in C++/CUDA. Run Llama 70B on RTX 3090.	2026-02-22
57	flash-tokenizer	461	9	C++	6	EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING	2026-02-02
58	zinc	452	18	Zig	3	Zig INferenCe Engine — Local LLM inference on AMD GPUs and Apple Silicon	2026-07-09
59	JetStream	451	66	Python	14	JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).	2026-01-05
60	Crane	422	45	Rust	14	A Pure Rust based LLM, VLM, VLA, TTS, OCR Inference Engine, powering by Candle & Rust. Alternate to your llama.cpp but much more simpler and cleaner..	2026-07-09
61	gpu-rest-engine	421	93	C++	6	A REST API for Caffe using Docker and Go	2018-07-20
62	FlashRT	409	52	C++	9	FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Al...	2026-07-10
63	AutoGrad-Engine	400	50	C#	0	A complete GPT language model (training and inference) in ~600 lines of pure C#, zero dependencies	2026-02-14
64	StockInference-Spark	382	193	Java	5	Stock inference engine using Spring XD, Apache Geode / GemFire and Spark ML Lib.	2016-06-03
65	sentis-samples	354	72	C#	11	Inference Engine samples internal development repository. Contains example and template projects for Sentis package use.	2025-11-06
66	flex-nano-vllm	353	20	Python	1	FlexAttention based, minimal vllm-style inference engine for fast Gemma 2 inference.	2025-11-02
67	InfiniTensor	346	69	C++	24	InfiniTensor is a high-performance inference engine tailored for GPUs and AI accelerators. Its design focuses on effective deployment and swift academic validation.	2026-06-11
68	rten	321	23	Rust	41	ONNX neural network inference engine	2026-07-09
69	AMDMIGraphX	314	140	C++	237	AMD's graph optimization engine.	2026-07-09
70	elfi	283	62	Python	10	ELFI - Engine for Likelihood-Free Inference	2025-05-07
71	yolov4-triton-tensorrt	283	61	C++	3	This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server	2022-06-02
72	awesome-edge-machine-learning	280	55	Python	1	A curated list of awesome edge machine learning resources, including research papers, inference engines, challenges, books, meetups and others.	2023-02-23
73	dash-infer	273	28	C	7	DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including CUDA, x86 and ARMv9.	2025-08-06
74	tflite2tensorflow	272	43	Python	1	Generate saved_model, tfjs, tf-trt, EdgeTPU, CoreML, quantized tflite, ONNX, OpenVINO, Myriad Inference Engine blob and .pb from .tflite. Support for building environments with Docker. It is possible ...	2022-09-04
75	mini-infer	271	13	Python	0	LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving	2026-04-24
76	whisper.el	263	23	Emacs Lisp	8	Speech-to-Text interface for Emacs using OpenAI's whisper model and whisper.cpp as inference engine.	2026-03-24
77	bw24	258	31	Rust	0	From-scratch Rust+CUDA inference engine, bit-exact by construction — NVFP4, MoE, MTP speculative decoding, tuned against measured limits of one RTX 5090 Laptop (sm_120a).	2026-07-10
78	compute-engine	256	35	C++	17	Highly optimized inference engine for Binarized Neural Networks	2026-06-26
79	oramacore	254	23	Rust	9	OramaCore is the complete runtime you need for your projects, answer engines, copilots, and search. It includes a fully-fledged full-text search engine, vector database, LLM interface, and many more u...	2026-04-14
80	inferflow	250	25	C++	8	Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).	2024-03-15
81	lm-inference-engines	240	9	-	8	Comparison of Language Model Inference Engines	2024-12-16
82	amd_inference	234	8	Python	11	Docker-based inference engine for AMD GPUs	2024-10-07
83	KokoroSharp	232	28	C#	18	Fast local TTS inference engine in C# with ONNX runtime. Multi-speaker, multi-platform and multilingual. Integrate on your .NET projects using a plug-and-play NuGet package, complete with all voices.	2026-04-11
84	Awesome-LLM-Inference-Engine	228	19	-	0		2026-06-30
85	ai-hardware-engineer-roadmap	224	36	HTML	0	Master AI inference, AI agent harness systems, and hardware engineering — then design a physical AI chip. That is the goal.	2026-06-25
86	MIVisionX	214	89	C++	8	AMD MIVisionX is a computer vision toolkit built around a highly optimized, conformant open-source implementation of the Khronos OpenVX™ 1.3 specification. As of the 4.0.0 release, MIVisionX ships thr...	2026-07-09
87	mlsub	203	21	OCaml	11	Prototype type inference engine	2025-01-31
88	embedded-ai.bench	201	29	Python	17	benchmark for embededded-ai deep learning inference engines, such as NCNN / TNN / MNN / TensorFlow Lite etc.	2021-02-18
89	TensorSharp	193	16	C#	1	A native .NET LLM inference engine for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It suppor...	2026-07-09
90	microflow-rs	185	27	Rust	3	A robust and efficient TinyML inference engine.	2026-05-26
91	RL-Kernel	182	48	Python	77	High-performance RL post-training infrastructure. Designed to achieve bitwise operator-level train-inference consistency across heterogeneous engines and extreme memory efficiency for GRPO, PPO, etc.	2026-07-09
92	rustane	177	22	Rust	3	Rust-native hybrid training & inference engine for Apple Neural Engine + Metal GPU	2026-04-03
93	world_engine	173	29	Python	12	World Model Inference Engine	2026-06-05
94	llm-systems-engineering-roadmap	167	24	-	0	A practical roadmap for mastering LLM internals, training, inference, RAG, agents, evaluation, and production architecture.	2026-04-25
95	fox	165	22	Rust	5	High-performance LLM inference engine — drop-in replacement for Ollama with faster multi-turn inference, lower TTFT, and higher throughput through prefix caching and continuous batching.	2026-07-08
96	TF2	164	60	Python	13	An Open Source Deep Learning Inference Engine Based on FPGA	2021-01-29
97	TurboLLM	164	27	TypeScript	2	Run any local LLM engine, auto-tuned to your GPU — polished web UI + OpenAI/Anthropic-compatible API. Point Claude Code at your own machine in one command. No Electron, no Python, offline-first.	2026-07-09
98	rf-detr-cpp	162	17	C++	0	Production-ready C++/TensorRT inference engine for RF-DETR. Object detection and instance segmentation with FP32/FP16/INT8 support. Optimized for NVIDIA GPUs, Jetson (Orin, AGX Thor).	2026-07-07
99	nanoPD	158	26	Python	1	A from-scratch Prefill/Decode disaggregation inference engine for LLMs	2026-05-10
100	zse	153	3	Python	1	The inference engine the open-source world built for itself.	2026-06-13